Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace threading.Lock with asyncio.Lock when batching to avoid deadlocks #1270

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open

Replace threading.Lock with asyncio.Lock when batching to avoid deadlocks #1270

wants to merge 19 commits into from

Conversation

tsmith023
Copy link
Contributor

@tsmith023 tsmith023 commented Aug 29, 2024 •
edited
Loading

  • Changes all threading.Locks in async def fns to asyncio.Lock to avoid blocking the event loop
  • Changes all time.sleep()s in async def fns to asyncio.sleep() to avoid blocking the event loop
  • Improves logging when errors occur during the batching logic
  • Uses BatchObject instead of _BatchObject in ErrorObject and BatchReference instead of _BatchReference in ErrorReference for easier usage of errors

@tsmith023 tsmith023 requested a review from dirkkul August 29, 2024 10:13
@codecov-commenter
Copy link

codecov-commenter commented Aug 29, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 87.50000% with 5 lines in your changes missing coverage. Please review.

Project coverage is 93.93%. Comparing base (3f68082) to head (feb148a).
Report is 166 commits behind head on main.

Files with missing lines Patch % Lines
weaviate/collections/classes/batch.py 82.35% 3 Missing ⚠️
weaviate/collections/batch/base.py 90.00% 2 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1270      +/-   ##
==========================================
- Coverage   94.38%   93.93%   -0.45%     
==========================================
  Files         220      221       +1     
  Lines       20956    21302     +346     
==========================================
+ Hits        19779    20010     +231     
- Misses       1177     1292     +115

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@weaviate-git-bot
Copy link

Great to see you again! Thanks for the contribution.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Slack channel?

@tsmith023 tsmith023 marked this pull request as draft September 16, 2024 16:45
@tsmith023 tsmith023 mentioned this pull request Oct 11, 2024
Open
@tsmith023
Copy link
Contributor Author

Closing in favour of #1371

@tsmith023 tsmith023 closed this Oct 30, 2024
@tsmith023 tsmith023 reopened this Nov 1, 2024
@tsmith023 tsmith023 mentioned this pull request Nov 1, 2024
Closed
@tsmith023 tsmith023 marked this pull request as ready for review November 1, 2024 11:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dirkkul dirkkul dirkkul approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@tsmith023 @codecov-commenter @weaviate-git-bot @dirkkul